Corpus Statistics Empowered Document Classification

In natural language processing (NLP), document classification is an important task that relies on the proper thematic representation of documents. Gaussian mixture-based clustering widespread for capturing rich semantics but ignores emphasizing potential terms in corpus. Moreover, soft approach causes long-tail noise by putting every word into cluster, which affects documents and their classification. It more challenging to capture semantic insights when dealing with short-length where co-occurrence information limited. this context, long texts, we proposed Weighted Sparse Document Vector (WSDV), performs weighted data emphasizes vital moderates removing outliers from converged clusters. Besides removal outliers, WSDV utilizes corpus statistics different steps vectorial document. For short Compact (WCDV), captures better building vectors uncertainty while measuring affinity between distributions words. Using available statistics, WCDV sufficiently handles sparsity texts without depending external knowledge sources. To evaluate models, performed a multiclass using standard performance measures (precision, recall, f1-score, accuracy) three long- two short-text benchmark datasets outperform some state-of-the-art models. The experimental results demonstrate long-text classification, reached 97.83% accuracy AgNews dataset, 86.05% 20Newsgroup 98.67% R8 dataset. 72.7% SearchSnippets dataset 89.4% Twitter